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', Abstract 

We present a generalization of multiple orthogonal polynomials of type I and type II, 
which we call multiple orthogonal polynomials of mixed type. Some basic properties are for- 
mulated, and a Riemann-Hilbert problem for the multiple orthogonal polynomials of mixed 
type is given. We derive a ChristofFel-Darboux formula for these polynomials using the solu- 
^ , tion of the Riemann-Hilbert problem. The main motivation for studying these polynomials 

comes from a model of non-intersecting one-dimensional Brownian motions with a given 
, number of starting points and endpoints. The correlation kernel for the positions of the 

Brownian paths at any intermediate time coincides with the Christoffel-Darboux kernel for 
the multiple orthogonal polynomials of mixed type with respect to Gaussian weights. 



o 

^ ■ 1 Introduction 

^ _ 

g ■ In the early nineties, Fokas, Its and Kitaev [T7j introduced a 2 x 2 matrix valued Riemann- 

Hilbert problem that characterizes orthogonal polynomials on the real line. This approach can 
be used to study various aspects of the theory of orthogonal polynomials. Combined with the 
^ ' Deift-Zhou steepest descent method for Riemann-Hilbert problems it has been very succesful 

. in deriving asymptotics for orthogonal polynomials and solving basic questions in the theory of 

random matrices ^3]. The Deift-Zhou steepest descent method was first introduced in and 
further developed in for example jl4j and jl5j . 

In , this Riemann-Hilbert problem was extended to the case of multiple orthogonal poly- 
nomials of type I and type II. Multiple orthogonal polynomials are polynomials which satisfy 
orthogonality conditions with respect to a number of measures. The definition will be given in 
section 2. The Riemann-Hilbert problem is now of size {p + 1) x {p+ 1), where p is the number 
of measures. 

In this paper, we introduce a generalization of multiple orthogonal polynomials. These new 
polynomials satisfy orthogonality conditions with respect to two sets of weights . . . ,uji^p 
and W2,i, ■ ■ ■ ,W2,q- We call these polynomials multiple orthogonal polynomials of mixed type. 
The definition will be given in section 2, together with some basic properties concerning existence 
and uniqueness. We can again characterize these polynomials by a Riemann-Hilbert problem 
which is now of size {p + q) x {p + q). This will be given in section 3. 

The usual monic orthogonal polynomials Pn on the real line with weight function w satisfy a 
three term recurrence relation and this gives rise to the basic Christoffel-Darboux formula (see 
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for example 



71—1 ^ 




1 P„(x)P„_i(y) - Pn-i{x)Pn{y) 



x-y 



(1.1) 



where 



/ij = / Pj{x)x^w{x)dx. 



(1.2) 



This formula was generalized to multiple orthogonal polynomials in Q in the case of two weights 
and in |T2I in the general case of p weights. We will derive using the Riemann-Hilbert problem 
a Christoffel-Darboux formula for the multiple orthogonal polynomials of mixed type. 

Our main motivation for studying this new kind of orthogonality comes from non-intersecting 
Brownian paths. Consider n independent one-dimensional Brownian motions that start in n 
different fixed points at time t = and end in n different fixed points at time t = 1, conditioned 
on the fact that they do not intersect in the full time interval (0, 1). At any intermediate time 
t E (0, 1) the positions of the Brownian paths are distributed according to a determinantal point 
process on the real line. This is a consequence of a classical theorem of Karlin and McGregor 
|20| and it applies not only to Brownian motion, but to any one-dimensional strong Markov 
process with continuous sample paths. 

Of special interest is the confluent case in which many of the starting points and many of the 
endpoints coincide. The formulas simplify for Brownian motion because of the properties of the 
Gaussian transition probabilities. In the extreme case of one starting point and one endpoint, 
the positions of the Brownian paths have the same distribution, up to simple scaling, as the 
eigenvalues of a matrix from the Gaussian Unitary Ensemble (GUE) which is a basic ensemble 
from random matrix theory. In this case the kernel for the determinantal point process is 
constructed out of Hermite polynomials and the Christoffel-Darboux formula (jl.lj) expresses 
this kernel in terms of Hermite polynomials of degrees n and n — 1 only. 

In the case of one starting point and q endpoints the positions of the Brownian paths have 
the same distribution as the eigenvalues of a Gaussian unitary matrix with external source 
[31 n I2E]- Then the kernel is constructed out of multiple Hermite polynomials of type I and 
II. The Christoffel-Darboux formula for multiple orthogonal polynomials of jl2j expresses this 
kernel in terms of a sum q + 1 terms, in which each term involves products of multiple Hermite 
polynomials of type I and type II. 

The next step is to consider Brownian motions with begin in p starting points and end in 
q endpoints. This gives rise to the multiple orthogonal polynomials of mixed type with respect 
to Gaussian weights. We will discuss this in more detail in section 6. Unfortunately we do not 
know if there exists a corresponding random matrix model 

For a summary of the above discussion, see Tabled 
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Non-intersecting 
Brownian motions 


Associated polyno- 
mials 


Random matrix en- 
semble 


1 starting point and 
1 endpoint 


Hermite polynomials 


Gaussian unitary 
ensemble 


1 starting point and 
q >2 endpoints 


multiple Hermite 
polynomials 


Gaussian unitary 
ensemble with 
external source 


p> 2 starting points 
and q > 2 endpoints 


multiple Hermite 
polynomials of 
mixed type 


unknown 



Table 1: Overview of the connection between non- intersecting Brownian motions, associated 
polynomials, and random matrix ensembles. The three cases are illustrated in Figures and 
13 See section 6 for a more detailed discussion. 

2 Multiple orthogonal polynomials of mixed type 

In this section we define the multiple orthogonal polynomials of mixed type and we give con- 
ditions for existence. These polynomials can be seen as a generalization of multiple orthogonal 
polynomials of type I and type 11 which we discuss first. Throughout this paper we will say that 
u) is a weight on M if w{x) > for x G M and f x'^w{x)dx < +cc for every /c G N U {0}. 

2.1 Multiple orthogonal polynomials of type I 

Let wi,W2, ■ ■ ■ ,Wp be p weights on the real line and let n = (ni,...,np) be a multi-index 
consisting of non-negative integers. If Ai, . . . , Ap are polynomials and 

p 

Q{x) = '^Aj{x)wj{x), deg Aj < Uj - 1, (2.1) 
such that 

jQ{x)x^dx = for j = 0, ...,|n| -2, (2.2) 

then the Aj are called multiple orthogonal polynomials of type I and Q is the linear form built out 
of the multiple orthogonal polynomials of type I. Here we follow the usual multi-index notation 

p 

1=1 

The relations 1)2. 2|) give us |n| — 1 homogeneous linear equations for the in total |n| coefficients 
of the polynomials Aj. So there is always a non-zero solution. If the solution is unique up to 
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Figure 1: Non intersecting Brownian motions which start and end at 1 point. At any interme- 
diate time the positions of the paths have the same distribution as the eigenvalues of a GUE 
matrix. The correlation kernel is built out of Hermite polynomials. 



a multiplicative factor, then the multi-index n is called normal for type I. The multi-index n is 
called strongly normal for type I if we have 



for any non-zero Q satisfying (|2.1j) - (|2.2j) . In that case we can normalize the multiple orthogonal 
polynomials of type I so that 

J Q(x)xl"l-idx = 1. (2.3) 

We call (|2.3|) a type I normalization. 

2.2 Multiple orthogonal polynomials of type II 

Let wi,W2, ■ ■ ■ ,Wq be q weights on the real line and let m = (mi, . . . , niq) be a multi- index of 
length q. If P is a polynomial of degree \rh\ such that 

J P{x)x^ Wk{x)dx = for j = 0, . . . , m^, — 1 and k = 1, . . . ,q, (2-4) 

then P is called a multiple orthogonal polynomial of type II. The \rh\ equations (|2.4|) are homo- 
geneous linear equations for the \rh\ + 1 coefficients of P. So there is always a non-zero solution. 
If the solution is unique up to a multiplicative factor then the multi- index \rh\ is called normal 
for type II. If every non-zero solution has a non-zero leading coefficient then the multi-index 
|m| is called strongly normal for type II. In that case we can normalize the multiple orthogonal 
polynomial of type II so that 

P„(x) = x" + --- (2.5) 

and we call (|2.5() a type II normalization. 

For more details and examples of multiple orthogonal polynomials of type I and type II, 
and about their relation with Hermite-Pade approximation, we refer the interested reader to 
[H 1211 127j and the references cited therein. 
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Figure 2: Non intersecting Brownian motions which start at 1 point and end at 2 different 
points. At any intermediate time the positions of the paths have the same distribution as the 
eigenvalues of a Gaussian unitary random matrix with external source. The correlation kernel 
is built out of multiple Hermite polynomials. 



2.3 Multiple orthogonal polynomials of mixed type 

To define the multiple orthogonal polynomials of mixed type, we need two sets of weights on R: 
■1^1,1) '"^1,2, ■ ■ ■ , wi,p and u'2,1, ■ ■ ■ , W2,q, which we collect in two row vectors 

Wl = {Wi^l, Wi^p), W2 = {W2,l, W2,q), 

and two multi-indices ft = (ni, . . . , Up) and m = (mi, . . . , niq) of length p and q, respectively, 
such that 

|n| = Ifhl + 1. (2.6) 

Definition 2.1 We call the polynomials Ai, . . . ,Ap with 

deg Aj <nj -1 for j = 1, . . . ,p (2.7) 

multiple orthogonal polynomials of mixed type for the pair of multi-indices (n, m) and with 
respect to the vectors of weights wi and W2 if the function 

p 

Q{x) = J2M^)^iA^) (2-8) 

satisfies the following orthogonality conditions: 

J Q{x)x^ W2,k{x)dx = for J = 0, 1, ... , m/j — 1 and k = 1, . . . ,q. (2-9) 
To emphasize the dependence on the multi-indices we also write 

■^j -^jfijThi Q Qn,rhj 

and to emphasize the role of the two vectors of weights we will occasionally write 

Aj{x) = Aj^fi^^{x;Wi,W2), Q{x) = Qft^rn{x;wi,W2), 
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Figure 3: Non intersecting Brownian motions which start at 2 points and end at 2 points. At 
any intermediate time the positions of the paths are distributed according to a determinantal 
point process with a kernel that is built out of multiple Hermite polynomials of mixed type. 



although mostly we drop the explicit mentioning of the weights. Note that the role of the two 
vectors of weights is not symmetric. The function Q is a linear form with respect to the weights 
from wi as in multiple orthogonality of type I, and the linear form has a number of orthogonality 
conditions with respect to the weights from ■W2 as in multiple orthogonality of type II. 

The conditions (|2.9)) lead to |m| homogeneous linear equations for the in total |n| free co- 
efficients of the polynomials Aj. Because of the assumption 1)2. 6() there is always a non-zero 
solution. If the polynomials Aj are unique up to a multiplicative constant, then we call (n, m) 
a normal pair of indices for the two sets of weights wi and W2 ■ 

For a normal pair of indices we can choose a certain normalization in order to define a unique 
multiple orthogonal polynomial of mixed type. In this paper, we are going to use two types of 
normalizations: 

• Type I normalization: Fix k = 1, . . . ,q and normalize Q such that 

j Q{x)x""=W2,kix)dx = 1. (2.10) 
If we choose this normalization, we write 

Aj = Af''\ g = g(^''=), (2.11) 

or in full notation, if we want to emphasize the dependence on the multi-indices and the 
weights 

Aj{x) = AfJ'l{x;wi,W2), Q{x) = Q^^'^^ {x;wi,W2). 

• Type II normalization: Fix k = 1, . . . ,p and normalize Q such that A^ is a monic 
polynomial of degree — 1. If we choose this normalization, we write 

Aj = Q = Q(''^^\ (2.12) 

or in full 

Aj{x) = A^^b^{x; wi,W2), Q{x) = Q^l^J^Xx; wi,W2)- 
We emphasize that the above normalizations may not always be possible. 
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2.4 Conditions for normality 

We will state the conditions using the Hilbert space geometry of L^(M) and to do so we assume 
that 

x^'tui^fe e l2(R), iov k = l,...,p, j = 0, 1,... - 1. (2.13) 

x^W2,k^ L^{^), fov k = l,...,q, j = 0,l,...,mfc-l (2.14) 

Associated with n and wi we have the vector space 

p 

Ffi = I is a polynomial of degree < nj — 1 for j = 1, . . . ,p} (2-15) 

i=i 

and associated with fh and W2 we have 
q 

Grn = ^j^2,j I Bj is a polynomial of degree < mj — 1 for j = 1, . . . , q} (2-16) 

i=i 

Hence -F^ is the linear span of the functions in 1)2. 13(1 and G^fi is the linear span of the functions 
in (tTTH) . 

Let e/c be the standard basis vector 

efc = (0, . . . , 0, 1, 0, . . . , 0), where 1 is in the kth. position. (2-17) 

We de not specify the length of the vector e^, but this should be clear from the context. 

Lemma 2.2 Suppose that Ffi is \n\- dimensional and let |n| = \m\ + 1. 

(a) Then Q is a linear form ()2.8() of multiple orthogonal polynomials of mixed type for the pair 
[ft, m) if and only if Q Ff^Ci . 

(b) The pair (n, rh) is a normal pair of indices if and only if Ffi Pi is one- dimensional. 

(c) The pair {n, m) allows a type I normalization with respect to the kth index of fh if and 
on/y «/F^nG4+.^ ={0}. 

(d) The pair (n, m) allows a type II normalization with respect to the kth index of n if and 
onlytfFn^^^nG^ = {0}. 

Proof, (a) This is immediate from the definitions. 

(b) If Ffi n is one-dimensional, then by part (a) the linear form Q is unique up to 
a multiplicative constant. Then the polynomials Aj are also unique up to a multiplicative 
constant, since Ffi is |n|-dimensional. Thus (n,rh) is a normal pair of indices. The converse is 
obvious in view of part (a). 

(c) Suppose Ffi n G^_^^^ = {0}. Let Q be a non-zero linear form of multiple orthogonal 
polynomials of mixed type. Then Q G G^ by part (a), but Q Since GfH C G^+^j, 
with codimension one, it follows that Q is not orthogonal to any function in G^f^^g^ \ G^ and 
since x'^^W2^k belongs to this set, this implies 

j Q{x)x'^'^W2,k{x)dx^Q. 

Then we can normalize Q so that this integral is 1 and the pair (n, fh) allows a type I normal- 
ization with respect to the kih. index. 
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On the other hand if n / {0}^ then any non-zero Q in this space would be a 

non-zero hnear form of multiple orthogonal polynomials of mixed type such that 

Then either (n, fn) is not normal, or if it is normal, it does not allow a type I normalization with 
respect to the kih index. 

(d) Suppose Ff(_g^ n = {0}. Let Q be a non-zero linear form of multiple orthogonal 
polynomials of mixed type for the pair (ra, m). Then Q £ Ffi hy part (a), but Q F^_g^. This 
implies that has exact degree n^ — l. Then we can normalize Q so that the leading coefficient 
of Aj. is 1 and thus the pair (n, m) allows a type II normalization with respect to the A;th index. 

On the other hand if Ffi_g^ n 7^ {0}, then any non-zero Q in this space would be a 
non-zero linear form of multiple orthogonal polynomials of mixed type for the pair (n, m) with 
the degree of A^ less than rik — l. Then either {n,rh) is not normal, or if it is normal, it does 
not allow a type II normalization with respect to the /cth index. □ 

We have the following easy corollary of Lemma 12.21 

Corollary 2.3 Suppose that 

(1) |n| = |m| and Ffi is a \n\- dimensional subspace o/L^(M). 

(2) F^nGi = {0}. 
Then the following hold: 

(a) For every k = 1, . . . ,q we have that {fi,rn — e^) is a normal pair of multi-indices which 

(I k) 

allows a type I normalization with respect to the kth index. Hence exists and is 

unique. 

(b) For every k = 1, . . . ,p we have that {n + ek,rn) is a normal pair of multi-indices which 

(II k) 

allows a type II normalization with respect to the kth index. Hence exists and is 

unique. 

Proof. Part (a) follows immediately from part (c) of Lemma 12.21 applied to the multi- indices 
n and m — e^), and part (b) follows from part (d) of Lemma 12.21 applied to the multi-indices 
n + efc and rh. □ 

3 The Riemann-Hilbert problem 

Fokas, Its, and Kitaev [IT found a Riemann-Hilbert problem that characterizes the orthogonal 
polynomials. Van Assche, Geronimo, and Kuijlaars (2Sj extended this Riemann-Hilbert problem 
to multiple orthogonal polynomials of type I and type II. We are now going to give a further 
extension to multiple orthogonal polynomials of mixed type. 

3.1 Riemann-Hilbert problem for multiple orthogonal polynomials of mixed 
type 

Assume that |n| = \m\ and let wi and W2 be as before. For convenience we assume that > 
and mi > for k = 1, . . . ,p and / = 1, . . . ,q. We can modify the arguments in case one or 
several of the and mi are zero, but we will not discuss that here. 

Consider the following Riemann-Hilbert problem: determine a (p + q) x (p + q) matrix valued 
function F : C \ M ^ c^p+q)>iip+q) such that 
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(1) y is analytic on C \ ] 

(2) for 2; G M, we have 

y+(x) =y_(x) 



p W{x) 



(3.1) 



where Ip and /„ denote the identity matrices of sizes p and g, respectively, and 



Wi^lW2,l W^l, 1^2,2 

W = W1W2 

_Wl,pW2,l Wi^pW2,2 

(3) as z ^ 00, we have that 

z'' 



Wl^lW2,q 
Wl^2W2,q 

Wl^p'W2,q 



(3.2) 



Y{z) 



Ip+q + O 























-mi 







where Ip+q denotes the identity matrix of size p + q. 



z- 



(3.3) 



As before we use and to denote the spaces (|2.15|) and (|2.16|) associated with the 
indices n and m. The main result of this section is that the Riemann-Hilbert problem has a 
unique solution if the conditions of Corollary 12.31 are satisfied. 

Theorem 3.1 Let \n\ = \m\. Suppose that Ffi is an \n\- dimensional subspace 0/ L^(R) such 
that Ffi n = {0}. Then the above Riemann-Hilbert problem has a unique solution, given in 
terms of the multiple orthogonal polynomials of mixed type with respect to the vectors of weights 
wi and W2 ■ We have 

• fork, 1 = 1,... ,p: 

• for k = 1, . . . ,q and I = 1, . . . ,p: 



(3.4) 



• for k = 1, . . . ,p and I = 1, . . . ,q: 

QHl£.Ax)w2,iix) 



{l,k) 



(3.5) 



Yk,p+i{z) = 
fork,l = l,...,q 
Yp+k,p+i{z) 



1 

2TTi 



X — Z 



dx, 



(3.6) 



dx. 



X — z 



(3.7) 
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Proof. First note that the functions Q^^^^g^^ and Q^fl'^_g^ uniquely exist ior k = 1, . . . ,p and 
/ = 1, . . . , g by Corollary 12.31 Partition Y as 



Yiz) 



K{z) Liz) 
M{z) N{z) 



(3i 



where K is a p x p matrix, L is a p x g matrix, M is a g x p matrix, and N is a q x q matrix. 
The jump condition 1)3. If) implies that 

K+iix) = K-p) for X e M, (3.9) 

so that K is analytic on the full complex plane. From (|3.9|) . the asymptotic condition (|3.3j) . and 
an extension of Liouville's theorem to polynomials, it then follows that each diagonal element 
Kk^ki with A; = 1, . . . ,p, is a monic polynomial of degree n^, and that each off-diagonal elements 
Kk^i is a polynomial of degree at most n; — 1. 

For the matrix L the jump condition (|3.1|) implies that for x G M, 

^k,li^) = ^~k,li^) + Qk{x)w2^l{x) for = 1, . . . ,p and ^ = 1, . . . , (3.10) 

where 

p 

Qk{x) = ^Kkj{x)wi^j. 
i=i 

The Sokhotsky-Plemelj formula then gives that 

L,K^) = — /^^^^^^^^x if.GC\M. (3.11) 
27ri ./ X — z 



If we now use the expansion 



z — X ^ — ' z'"^-" Z" Z — X 
1=0 



we find that 
LkAz) 



1 r 

j=0 

fQM^yiMd, fork = l,...,pandl = l,...,q. (3.13) 

J z-x 



2'iriz"- 

The asymptotic condition 1)3. 3() gives that 

lim LkAz)z^^ = 0, (3.14) 

such that from ()3.13|) with n = mi-i we get 

J Qkix)x-'w2,i{x)dx = for j = 0, ... ,mi — l,k = 1, ... ,p and I = 1, ... ,q. (3.15) 

Because K^^i is a polynomial of degree at most — 1 if / 7^ A; and K^^k is a monic polynomial 
of degree n^, as mentioned in the beginning of the proof, and because of (|3.15jl we see that 

Kfc; = 4''^^ ^ (3.16) 
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ior k,l = 1, . . . ,p. Because of (|3.1ip and (|3.16|) we see that 

1 f QH+etrni^)^2,li^) 



Lkl(z) = / "^"'-''"^ '—dx ifzGCVM (3.17) 

2m J X — z 

for /c = 1, . . . ,p and / = 1, . . . , q. This proves the formulas (|3.4)) and (j3.6|) . 
The jump condition (|3.1j) shows that for A; = 1, . . . , g and I = 1, . . . 

M+ (x) = M,-;(x) for X G M, (3.18) 

and consequently M is also analytic on the full complex plane. In the same way as for the matrix 
K, the asymptotic condition (|3.3j) implies that each M^^i is a polynomial of degree < n/ — 1. 
The jump condition (|3.1|) also implies that 

N+iix) = N^^lix) + Qk{x)w2,i{x) for /c,/ = 1,... ,g and X G M, (3.19) 

with 

p 

Qk{x) = y^Mfcj(x)wij(x). 

i=i 

Using the Sokhotsky-Plemelj formula we get that 

Ar^^(^) = ^ f Qk{x)w2,i{x) for A;,/ = l,...,gif zgC\M. (3.20) 
27ri y x — z 

The asymptotic condition (|3.3jl gives that 
lim Nki{z)z'^' =0 if A;//, 

2:— ►oo ' 

lim iVfc,fc(z)z"^'= = 1, 

z~*oo 

Using these conditions and the expansion (|3.12|) we get that 

/ Qk{x)x^W2^i{x)dx = for j = 0, 1, . . . , — 1 if A; / /, 
* J Qk{x)x^W2,k{x)dx = for j = 0, 1, . . . ,mfc - 2, (3.21) 
f Qk{x)x''^''^^W2^k{x)dx = — 27ri. 



The degree of the polynomials Mk^i and the orthogonality conditions (|3.21|) imply that Mk^i 

l,n,rn—ek 



-2'KiA9^\_g for A; = 1, . . . , g and I = 1, . . . ,p. Because of (|3.2U() we get that 



Nk,ii^) = - — ^73^ dx (3.22) 

for k,l = 1, . . . ,q and z £ C\M. This proves the formulas 1)3. 5() and (|3.7() . This completes the 
proof of Theorem l3.11 □ 
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3.2 Riemann-Hilbert problem for the inverse 

By standard arguments it follows that det Y(z) = 1 for z € C \ M so that the inverse Y~^{z) 
exists and is analytic for z £ C \ M. Define 

X{z) = Y'\z). (3.23) 

From the Riemann-Hilbert problem for Y it is then straightforward to check that X is the 
solution of the following {p + q) x {p + q) matrix valued Riemann-Hilbert problem: 

(1) X : C \ M ^ ^{p+q)x{p+q) -g analytic, 

(2) for X G M, we have 



X+{x) = X^{x) 



Ip 

-W\x) I, 

where W is given by (|3.2|) . 
(3) as z ^ oo, we have that 



X(z) 



'p+q 



+ 0{- 

z 



-ni 



Z 






-"2 






























z^^i 



(3.24) 



(3.25) 



The solution of the Riemann-Hilbert problem for X can again be written in terms of multiple 
orthogonal polynomials of mixed type, but with the roles of the vectors of weights wi and 
W2 as well as the multi-indices n and m interchanged. Therefore we use the full notation 

Lemma 3.2 Suppose that the conditions of Theorem \'A.l\ are satisfied. Then the solution of the 
above Riemann-Hilbert problem has a unique solution given by 



For k,l = 1, . . . ,p: 



Ql^,'?-e-,(^;^2,^l)wi,;(x) 
X — Z 



dx, 



(3.26) 



for k = 1, . . . ,q and I = 1, . . . ,p: 

^ . ^ 1 /• QS£,n(^;^2>i)wM(a:) 

x,+,A^) = -— / dx, 



(3.27) 



for k = 1, . . . ,p and I = 1, . . . ,q: 



Xk^p+i{z) = 2TTiA 



{i,k)^ 

Lfh,n—ei 



{z;w2,wi), 



(3.28) 



for k,l = l,...,q: 

Xp+k,p+iiz) = A[^^i;ll^^^{z;w2,wi] 



(3.29) 
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Proof. The lemma can be proven in the same way as Theorem 13.11 but it is also possible to 
derive it directly from Theorem 13.11 as follows. Let U be the solution of the Riemann-Hilbert 
problem described in Theorem 13.11 but with the roles of the vectors of weights wi and W2 as 
well as the multi-indices n and m interchanged. By comparing the jump conditions and the 
asymptotic conditions of the Riemann-Hilbert problem of X and U, we can easily see that 



Uiz) 







-I, 




X{z) 



Ip 

-la 



(3.30) 



Theorem 13. II therefore implies that the solution of the Riemann-Hilbert problem defined above 
is unique and is given by the formulas (|3.26j) ~ (|3.29j) . □ 



4 The kernel 

Suppose that wi = {wi^i, wi^2, ■ ■ ■ , and 102 = (11^2,1, W2,2, ■ ■ ■ , W2,q) are two vectors of weights 
on the real line, and define and as in (|2.15|) and (|2.16|) . Suppose as in section 3 that 
\n\ = \m\ = n and that Ffi and are both n-dimensional subspaces of L-^(]R). Two bases 
(pi, . . . , (pn of Ffi^ and ipi, . . . ,ipn of G^f^ are called biorthogonal if 



/ 



(j)j{x)il;kix)dx = 6j^k- 



The following lemmas are well-known, but we include their proofs for completeness. 
Lemma 4.1 There exist biorthogonal bases for Ffi and if and only if Ffi n G^ = {0}. 

Proof. First suppose that 0i, . . . , 0„ G and ^i, . . . ,ipn G are biorthogonal bases. Then 
every f £ Ffi can be written as 

n 

f = Y.^3<Pj (4-1) 

with 

f{x)^j{x)dx, j = l,...,m. (4.2) 

Since every Cj = if / e G^ it follows that F^; n G^ = {0}. 

Conversely, suppose that n G^ = {0}. Let V'l, • • • , V'n be any basis of G^j. Consider the 
linear mapping 

T-.Fn^W'-.f^iJ f{x)Mx)dx)k=i,...,n. (4.3) 

Because F^ n G^ = {0} it is clear that is injective. Since the dimensions of M" and F^ are 
equal, and the mapping is linear, T is bijective. Consequently there exist functions cpj S F^, j = 
1, . . . , n such that J (f)j{x)'ipk{x)dx = 5j^k- Then (pi, ...,(/>„ is a basis of F^ which is biorthogonal 

to Ipn, - ■■ ,1pn- n 

From now on, we assume that \n\ = \m\ and F^ n G^ = {0}. According to Lemma l4. 1 1 there 
exist biorthogonal bases (pi, . . . , (pn oi Ffi and ipi, . . . ,ipn of G^fi. We define the kernel K{x, y) as 

n 

K{x,y) = Y,Ux)^j{y)- (4.4) 
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Lemma 4.2 The kernel K{x,y) is the kernel of the (non- orthogonal) projection operator onto 
Ffi parallel to G^. 



Proof. Define the operator K on L' 



as 



{Kh){x) 



K{x,y)h{y)dy. 



(4.5) 



By the definition (|4.4j) of the function K{x,y) it is clear that Kh = if /i G G^. In the same 
way it is obvious that Kh = h h = (p/., for k = 1, . . . ,n. By hnearity it then follows that 
Kh = h for every h G Ffj. Because © = -L^(M), Lemma 14.21 follows immediately. □ 

Lemma 14.21 also implies that the kernel K is independent of the chosen biorthogonal bases. Now 
we arrive at the main result of this paper: 

Theorem 4.3 The kernel K{x,y), as defined in (|4.4I) . can be written in terms of the solution 
of the Riemann-Hilbert problem for Y of section 3.1 in the following way: 



K{x,y) 



2'Ki{x — y) 



[0 



W2,i{y) ■■■ W2,M]Y+\y)Y+{^) 



wi^i{x) 




(4.6) 



Proof. Define the operator L on L^(R) as 



{Lh){x) 



L{x,y)h{y)dy, 



(4.7) 



where L{x, y) denotes the right-hand side of 1)4. It is enough to prove the following two things: 

(a) L/i = if /i G , 

(b) Lh = hiih£ F^. 

Indeed, if this is the case, then L is the projection operator onto -E^ parallel to G^, and according 
to Lemma we get that K{x, y) is the kernel of L, and consequently K{x, y) = L{x, y). 

(a) First let h G G^. For ease of notation we will use [t(;i(x) O] and [o W2{y)\ instead of 
[tDi^i(x) ••• wi^p{x) ••• O] and [O ••• ^2,1(2;) ••• ^1)2,^(2;)] , where in each case 
the number of zeros is such that the length of the vectors \s p + q. We then have that 

^^^^^^^ =2^ / ^^y^ [° ^2^^^] ^^^^^4^^^^+^^^ [^^^"^^ 



+ -i, / h{y) [0 W2{y)] ^^^Y+{x) [w^{x) Q\'dy. 
iTii J '- ^ X — y 



(4.8) 



Because Y]^^(x)Yj^{x) = / and [() t?2(y)] [^^1(3;) O]* = 0, the second term of (|4.8j) is equal to 
zero. The form of the solution of the Riemann-Hilbert problem for X = Y~^ as given by ()3.28|) 
and (|3.29|) implies that the last q rows of 



Y^\y)-Y^\x) 



x-y 



(4.9) 
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deg 



< mj — 1. 



consist of polynomials in the variable y such that for j = 1, . . . , g, and k = 1, . . . ,p + q, 

X — y 

J p+j,k 

This implies that for each fixed x E M, each entry of the row vector 

Y-\y)-Y-\x) 



(4.10) 



[0 W2{y)] 



x-y 



belongs to G^. Because h G G^, the first term of H4.8() is equal to zero as well. Thus Lh = 
and this proves (a). 

(b) Now let h £ F^. Then h = X]j=i where Aj is a polynomial of degree less 

than or equal to nj — 1. We write ^ = (^1, ^p) and A{x) = [^i(x) ••• Ap{x) ••• 
We then have that h{x) = A{x) O]* and so 



iLh){x) 



2m 



A{y)-A{x) 



x-y 



[wi{y) ^*[0 W2{y)\Y+\y)y+{^)[wi{x) ^'dy 



+ 



2m 



A{x) Q\[wi{y) 0]*[0 W2{y)]Y~\y)^±^[w^{x) 0]*dy. 
J X y 



(4.11) 



We will deal first with the first term in the right-hand side of ()4.11[) . We have the combination 
[O W2{y)] Y^ (y) which is a row vector whose kth entry is 



(^+ ^)p+i,k (y)^2,/(2/) = ^Xk,p+iiy)w2,iiy) 



1=1 



1=1 



since Y ^ = X^, see 1)3. 23() . The functions are certain multiple orthogonal polynomials 

of mixed type given explicitly by formulas (|3.28|) and (|3.29j) . Then it follows that 



([0 W2{y)] Y^Hy))^ = 2mQll'^l,^iy;w2,wi) iork = l,...,p 



and 



([0 My)] y+'iy)),^, = Qli';tn(y'^2,w,) for k = i,...,q. 



(4.12) 



(4.13) 



Since 

for each fixed x S M, 



is a polynomial of degree < nj — 2 in the variable y for j = 1, . . . ,p, we have 



A{y)-A{x) 



x-y 



(4.14) 



From the defining properties of the multiple orthogonal polynomials of mixed type, it follows 
that each of the functions ()4.12|) and ()4.13() is orthogonal to -F^_^p ^ . Then it follows that 
the first integral in the right-hand side of (|4.11|) is zero for every x. 

Now we come to the second term in the right-hand side of ()4.11|) . We are going to show that 
for every x G M, 



^J[wi{y) 0t[0 W2{y)]Y-\y)^dy 



Ip * 

* * 



(4.15) 



15 



where * represents an unspecified unimportant entry (which may actuahy be a divergent inte- 
gral). Having ()4.15|) we easily see that the second term in the right-hand side of ()4.11|) reduces 

to A{x) O]* = h{x), independent of what the unspecified entries are (even if they 

are divergent integrals). 

In order to establish (|4.15|) we note that the jump condition p.lj) written in the form 



y+(y) = y_(y)(/+[u;i(y) O] [O W2{y)\ 
implies that 

[wi{y) 6]*[0 W2{y)]Y^\y) =Yl\y)iY+iy) -Y^iy))Y^\y) 



Yl\y)-Y-\y). 



Thus the left-hand side of (|4.15|) is 



1 

27ri 



Yl'{y)-Y^'{y) 



x-y 



Y+{x)dy 



(4.16) 



(4.17) 



Let z G C with Imz > 0. Then Y ^{y)/(z — y) is analytic in the lower half plane and from the 
Riemann-Hilbert problem satisfied by X = Y~*, it follows that for k,= 1, . . . ,p, I = 1, . . . ,p + q, 



[Y-Hy)] 



kl 



0{y 



z-y 

It follows that (here we use > 1 



as y ^ OG. 



(4.18) 



1 

27ri 



z-y 



dy = 0, for A; = 1, . . . ,p, l = l,...,p + q. 



(4.19) 



Similarly, we have that Y~^{y)/{z — y) is analytic in the upper half plane but with a pole at 
y = z. Then if we calculate the same integral as in ()4.19|) but with Y_ replaced by y+, and we 
use the decay property (|4.18|1 . the only contribution comes form the residue sX y = z and the 
result is 



1 



kl 



dy 



Y- 



z-y 

From KT^ and KTHii it follows that 
Yl\y)-Y-\y) 



H^))k,r 



for k = 1, 



,P, 



l = h 



(4.20) 



27ri 



Letting z ^ x £ 



z-y 
it follows that 



dy Y{z) 



Ip * 



for Imz > 0. 



(4.21) 



1 

27ri 



Y~\y)-Y^\y) 



x-y 



Y+{x)dy 



Ip * 



for X € 



(4.22) 



which implies (|4.15|) by (|4.16|1 . As noted after (|4.15|) it then follows that the second term in the 
right-hand side of ()4.11|) is equal to h{x). Since we already know that the first term is equal to 
0, we have proved that Lh = h. This completes the proof of Theorem 14., SI □ 
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5 The ChristofFel-Darboux formula 



Theorem l4.3l imp lies a Christoffel-Darboux formula for multiple orthogonal polynomials of mixed 
type. We assume as before that \n\ = \rn\ and that Ffi and G^fi are both |n| -dimensional subspaces 
of L2(M) such that FfinGj^ = {0}. 

Corollary 5.1 Let K he the kernel defined in ()4.4() . We can write the kernel in terms of the 
multiple orthogonal polynomials of mixed type defined in H2.11() and H2.12() as follows: 

p 

{X - y)K{x, y)=Y, Qnlilrni^'^ Wi,W2)Qll'^J_g^ {y, W2,Wi) 

i=i 



J2 QhS-sS"^'^ wi,W2)Q^i^^%^f.{y; W2,wi). (5.1) 



k=l 

Proof. The kernel K and the multiple orthogonal polynomials of mixed type are well defined 
because of Corollary 12.31 and Lemma l4.ll The entries of Yj^[x) in the first p columns are given 
by (1231) and (ESJ. The entries of y+ ^y) = X\y) in the last q rows are given by (|3.28|) and 
(|3.29jl . Inserting these formulas into (|4.6|) we arrive at (|5.1j) . □ 

Remark 5.2 The usual monic orthogonal polynomials on the real line with weight function 
w{x) satisfy the classical Christoffel-Darboux formula (|1.1() . By putting p = q = \ m. formula 
(|5.H) and taking into account that the type II normalization for the multiple orthogonal polyno- 
mials of mixed type is different from the normalization used for monic orthogonal polynomials, 
we can see that (|5.1|) reduces to (|1.1|) in case p = q = 1. 

In [i^ the special case p = 1 and q = 2 is considered in connection with random matrices 
with external source. This leads to a kernel built out of multiple orthogonal polynomials with 
respect to 2 different weights for which a Christoffel-Darboux kernel was given. In 12^ the 
Christoffel-Darboux formula was generalized to multiple orthogonal polynomials with respect to 
q different weights: 

{x - y)K{x,y) = PH{x)QH{y) - ^ ^^PH-e,{x)Qn+eM- (5-2) 

k=l ^n-Cfc 

Here is the multiple orthogonal polynomial of type II, is the linear form constructed out 
of the multiple orthogonal polynomials of type I, with the type I normalization as described in 

ik') (k) 

section 2, and the h\ and h-._^ are certain constants. The formula 1)5. 2|) is the special case 
p = lof 

Remark 5.3 When we take the multi-indices n and m in a way such that 

ni < n2 < ■ ■ ■ < np < ni + 1 and mi < m2 < • • • < rUg < mi + 1, (5-3) 

then the multiple orthogonal polynomials of mixed type are vector polynomials orthogonal with 
respect to the weight 



W{x) 



Wl,lix)w2,lix) Wl^l{x)w2,2ix) ■■■ Wl^l{x)w2,q{x) 
Wl,2{x)w2,l{x) Wl^2ix)w2,2ix) ■■■ Wl^2{x)w2,q{x) 

Wl,p{^)w2,l{x) Wl^p{x)'W2,2ix) ••• Wl^p{x)'W2,q{x)^ 



(5.4) 



In [25, Sorokin and Van Iseghem obtained a Christoffel-Darboux formula for vector polynomials. 
Their formula has ^^^^^^ -\- '^^'^^^^ terms, while ours has only p + q terms. 
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6 Non intersecting Brownian motions 



Our motivation for introducing the multiple orthogonal polynomials of mixed type came from the 
theory of non-intersecting Brownian motions. Consider n one-dimensional Brownian motions 
which start at n fixed points ai < a2 < ■ ■ ■ < an &t time t = and end at n fixed points 
61 < 62 < • • • < 6„ at time t = 1. Let Pn,t{xi, • • • , x^) denote the probability density that at 
time t, with < t < 1, the paths are at the positions xi, . . . conditioned on the event that 
the paths do not intersect in the full time interval (0, 1). Then it follows from a result of Karlin 
and McGregor [20] that 

Pn,t{xi,...,Xn) = ^det(P(i,aj,Xfc))";,=idet(P(l - t,bj,Xk))'j^k=i> (6-1) 



where 



P(t, a, x) = ^=e-5?(^-")' (6.2) 

is the transition probability for the one-dimensional Brownian motion and Z„ is a normalization 
constant. Note that ()6.1|) is an example of a biorthogonal ensemble iTj. 

Consider now the confluent case that some of the starting points and some of the end points 
coincide. Suppose that the n non-intersecting Brownian motions start at p different points aj, 
j = 1, . . . ,p, where aj appears with multiplicity rij, and end at q different points bj, j = 1, . . . ,q, 
where bj appears with multiplicity rrij. Let n = (ni, . . . , Up) and fh = (mi, . . . , niq) and 

p 

Ffi = {S^Aj{x)P{t,aj,x) I Aj polynomial with deg(j4j) < nj — 1} (6.3) 



and 



= {"^^ Bj{x)P{l — t,bj,x) I Bj polynomial with deg{Bj) < mj — 1}. (6.4) 

So these are the spaces (|'2.15|) and (|2.1H|) associated with the vectors of weights wi = {wi^i, . . . , tfi,p) 
and ^2 = ('"^2,1 5 • • • , W2,p) where 

wij{x) = P{t,aj,x), for j = 1, . . . ,p, 

and 

W2,j{x) = P{1 - t,bj,x), for j = 1, . . . ,g. 

Lemma 6.1 The spaces Ffi and Gfj^ are n- dimensional and 

F^nG4,={0}. (6.5) 

Proof. The statement about the dimensions are obvious. The proof of (|6.5() is based on the 
following facts: 

(1) Both Ffi and are Chebyshev spaces on M, which means that any non-zero function in 
one of these spaces has at most n — 1 zeros on R, see e.g. To show this, we note that 
by an example given in {211 Chapter 4, §4], the functions 

g/3ia;^ . . . , x"'^~^e^^^, . . . , e^p^, . . . , x"'''~^e^p^ (6-6) 

form a Chebyshev system of order n — 1 on M whenever Pi, . . . , f3p are distinct real numbers. 
Taking f3j = ^Oj, and multiplying the functions (|6.6() by the common factor e~2t^ , we 
obtain a basis of -F^, and so Ff^ is a Chebyshev space on M. Similarly we have that is 
a Chebyshev spaces on M. 
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(2) For any set of distinct real points xi, . . . , x^ with m < n — 1, there exist functions / G Ffi, 
g G G^, such that / and g change sign exactly at each of these points. This is a general 
property of Chebyshev spaces, see |H1 Chapter 3.1, exercise E.ll]. 

Now let f G Ffi he non-zero. Then / has at most n — 1 real zeros of by (1). Let xi, . . . , Xm 
with m < 71 — 1 be the zeros of odd multiplicity (so that / has a sign change at these points). 
By (2) there is € which also changes sign exactly at these points. Then fg has no sign 
change on M, and therefore / f{x)g{x)dx > 0. Thus / does not belong to G^-t and (|6.5|) follows. 

□ 

By Lemmas 15.11 and 14.11 there exist biorthogonal bases . . . ,(pn of -F^ and ipi, . . . ytpn of 
Ga. Let Kn be the projection kernel 

n 

i=i 

Then in the confluent case the probability density 1)6. 1|1 to find the Brownian paths at time t at 
the positions xi, . . . ,Xn can be written as 

p„,t(xi, . . . , a;„) = det(0j(xfc))"^fc=i det(V'j(2;fc))"fc=i = ^ det{Kn{xj,Xk))lk=i. 
Moreover, all correlation functions have determinantal form with kernel K„- That is, if 



n 



• • • 1 — / M I ' ' ' I Pn.t{xi , . . . , S^m; Xm+1 1 • • • Xn^dXm+1 ' ' ' dXj-i 

{n-m)\ J ^ J ^ 
n-m times 

denotes the m-point correlation function, then 

fmixi^ . . . , Xm) — det{Kn{Xj , Xfc))j^^_]^ 

for every m = 1, . . . ,n. 

There is a substantial literature on determinantal point processes and non-intersecting ran- 
dom paths see e.g. the recent surveys ^1 El H5] and references cited therein. 

Our Theorem l4. 31 relates the kernel Kn to the Riemann-Hilbert problem for multiple orthog- 
onal polynomials of mixed type. This opens up the possibility to analyze the kernel in the large 
n limit with the Deift/Zhou steepest descent method for Riemann-Hilbert problems. 

To obtain interesting limit behavior, one first modifies the transition probability (|6.2|) to 

P„(t,a,x) = ^e-i(--'^)' (6.8) 
V zvrt 

so that the overall variance of the Brownian paths is reduced with increasing n. With increasing 
n, the starting points ai, . . . , Op and the end points bi, . . . ,bq remain fixed while the correspond- 
ing multiplicities ni, . . . , rip and mi, . . . , niq increase with n, such that the limits 

lim — and lim — - 

n-^oo fi n— >oo 77, 

exist and are positive. 

Based on the experience with the case p = 1 and q = 2 that was developed in j^l El El 
we expect the following to hold true. In the limiting regime described above, with probability 
one, the Brownian paths fill out a bounded region as illustrated in Figure |3 Figure |11 shows a 
possible boundary curve in the case of two starting points and two endpoints. Here we see that 
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Figure 4: Non intersecting Brownian motions which start and end at two different points 



two groups of paths start from two different starting points and they come together and merge 
at a certain critical time. Then they continue as one group until at a second critical time they 
split again into two groups that end at the two different end points. The boundary curve is 
smooth except for cusp singularities that arise when two groups of paths come together or split. 
We expect that this behavior is generic for general p and q. 

At any time t £ (0, 1) we further expect that the correlation kernel Kn has a scaling limit 
which is equal to the usual scaling limits from random matrix theory. That is, if we scale around 
a point (t, x) lying strictly inside the boundary curve then we expect the sine kernel in the limit 
and for a usual point (t, x) on the boundary (not a cusp point) we expect the Airy kernel. 

At the cusp singularities we expect that the kernel Kn has the Pearcey kernel as a double 
scaling limit. This Pearcey kernel arose first in the works of Brezin and Hikami jHl El in the 
context of Gaussian random matrices with external source. In our notation this corresponds to 
p = 1 and q = 2. A detailed treatment based on a double integral representation of the kernel 
was made by Tracy and Widom . These authors also considered an extended Pearcey kernel 
and a Pearcey process which involves the limiting joint distributions at several scaled times near 
the critical times. The Pearcey process also appears in the recent papers PI \'2'M . 

For an extension of the above results to more general values of p and q the Riemann-Hilbert 
problem that we gave in this paper might be useful. Indeed, if the Deift/Zhou steepest descent 
analysis can be made to work on this Riemann-Hilbert problem then the scaling limits of the 
kernel can be derived. We plan to report on this in a later publication. 
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